Hybrid image/scene renderer with hands free control

ABSTRACT

A system and method for converting static/still medical images of a particular patient into dynamic and interactive images interacting with medical tools including medical devices by coupling a model of tissue dynamics and tool characteristics to the patient specific imagery for simulating a medical procedure in an accurate and dynamic manner by coupling a model of tissue dynamics to patient specific imagery for simulating surgery on the particular patient. The method includes a tool to add and/or to adjust the dynamic image of tissues and ability to draw any geometric shape on the dynamic image of tissues and to add the shape into the modeling system.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/402,746 filed on Nov. 21, 2014 and incorporated herein by reference,which is the national phase of International Application No.PCT/US2013/42654 filed on May 24, 2013, the entire disclosure of whichis incorporated herein by reference, which claims the benefit of U.S.Provisional Patent Application Ser. No. 61/651,775 which was filed onMay 25, 2012 and is incorporated herein by reference

BACKGROUND

This application relates generally to a system and method for simulatingsurgical procedures. More specifically, this application relates to asystem and method for converting static/still medical images intodynamic and interactive images interacting with medical tools (such as,e.g., surgical tools, probes, and/or implantable medical devices) bycoupling a model of tissue dynamics to patient specific imageryutilizing hands-free control.

Surgeons lack a rehearsal and preparation tool that would provide themwith a realistic visual model with physical tissue properties. Mostimportantly, it is desired to have a “full immersion” surgical tool thatencompasses: (i) realistic “life-like” 2D and/or 3D display of thepatient-specific area of surgery (for example—aneurysm); (ii) modelingof the local patient-specific area of surgery geometry and physicalproperties; (iii) interface enabling manipulation of thepatient-specific area of surgery model and virtually perform surgicalactions such as cutting, shifting and clamping; and (iv) interface toprovide feedback cues to the surgeon.

Furthermore, tools that allow the surgeons to perform simulations usinga hands-free control would be useful as well, along with means forcorrecting deficiencies in, or otherwise modifying, the graphical imagesof the tissue models.

SUMMARY

Provided are a plurality of example embodiments, including, but notlimited to, a modeling system for performing a medical procedure,comprising: a display; an image generator for generating a dynamic imageof tissues for display on the display, the generating for displaying onthe display the tissues realistically representing corresponding actualbiological tissues; a user tool generator for generating a tool model ofa user tool for dynamically interacting with the dynamic image oftissues via manipulations provided by a user input for display on thedisplay; and a user interface providing a tool to adjust the dynamicimage of tissues displayed on the display by adding or modifyingfeatures of the tissues to compensate for anatomical structures that arein the actual biological tissue but are missing from the dynamic imageof tissues originally displayed such that the dynamic image of tissuesdisplayed are subsequently displayed on the display with the added ormodified features. The tool model is displayed on the displaydynamically interacting with the dynamic image of tissues forrealistically simulating the medical procedure.

Also provided is a modeling system for enabling a user to perform asimulated medical procedure, the system comprising: one or morecomputers; a display for displaying images to the user; a database forstoring physical characteristics of the tissues of a particular patient;an image generator using one or more of the computers for executingsoftware for generating a dynamic realistic image of the tissues of theparticular patient for displaying on the display, wherein the realisticimage of the tissues is provided showing an appearance includingshadowing and textures indicative of the tissues of the particularpatient; a user tool library for providing a plurality of user toolmodels of actual user tools used in medical procedures; a user interfacefor accepting inputs from the user for selecting one of the user toolmodels; a user tool generator using one or more of the computers forexecuting software for generating a realistic tool image of the selecteduser tool model for displaying on the display; a user interface foraccepting inputs from the user, the inputs for dynamically manipulatingthe selected user tool image for dynamically interacting with therealistic image of the tissues during the simulated medical procedurefor display to the user on the display in real-time; and a userinterface providing a tool to adjust the dynamic image of the tissuesdisplayed on the display by adding or modifying features of the tissuesfor display to compensate for anatomical structures that are in theactual biological tissue of the particular patient but are missing fromthe dynamic image of tissues originally displayed such that the dynamicimage of tissues displayed are subsequently displayed on the displaywith the added or modified features. The dynamic interaction between theuser tool image and the image of the tissues is displayed on the displayusing images with realistic visual features exhibiting realisticmechanical interactions based on the stored physical characteristics.

Further provided is a modeling system for performing a surgicalsimulation, comprising: a database for storing patient tissue imageinformation that are taken from, or derived from, medical images of aparticular patient; the database also for storing standardcharacteristics of the tissue; a display; an image generator forgenerating a dynamic image of tissues of the particular patient fordisplay on the display, the generating utilizing the patient imageinformation such that the dynamic image of tissues is displayed on thedisplay realistically representing corresponding actual tissues of theparticular patient; a user tool generator for generating a tool model ofa user tool for dynamically interacting with the dynamic image oftissues via manipulations provided by a user for display on the display;and a user interface providing a tool to adjust the dynamic image oftissues displayed on the display by adding or modifying features of thetissues for display to compensate for anatomical structures that are inthe actual biological tissue of the particular patient but are missingfrom the dynamic image of tissues originally displayed such that thedynamic image of tissues displayed are subsequently displayed on thedisplay with the added or modified features. The tool model is displayedon the display dynamically interacting with the dynamic image of tissuesfor realistically simulating the medical procedure.

Also provided is a modeling system for enabling a user to perform asimulated medical procedure, the system comprising: one or morecomputers; a display for displaying images to the user; a database forstoring characteristics of the tissues of a particular patient; an imagegenerator using one or more of the computers for executing software forgenerating a dynamic realistic image of the tissues of the particularpatient based on the stored characteristics of the particular patientfor displaying on the display, wherein the realistic image of thetissues is provided showing an appearance including shadowing andtextures indicative of the tissues of the particular patient; a usertool library for providing a plurality of user tool models of actualuser tools used in medical procedures; a user interface for acceptinginputs from the user for selecting one of the user tool models; a usertool generator using one or more of the computers for executing softwarefor generating a realistic tool image of the selected user tool modelfor displaying on the display; and a user interface including a camerafor accepting hands-free inputs from the user, the inputs fordynamically manipulating the selected user tool image and/or the imageof the tissues for dynamically interacting with the realistic image ofthe tissues during the simulated medical procedure for display to theuser on the display in real-time. The dynamic interaction between theuser tool image and the image of the tissues is displayed on the displayusing images with realistic visual features exhibiting realisticmechanical interactions.

Also provided is modeling system for enabling a user to perform asimulated medical procedure, the system comprising: one or morecomputers; a display for displaying images to the user; an imagegenerator using one or more of the computers for executing software forgenerating a dynamic realistic image of the tissues for particularpatient for displaying on the display, wherein the realistic image ofthe tissues is provided showing an appearance including shadowing andtextures indicative of actual tissues; a database for storing a usertool library for providing a plurality of user tool models of actualuser tools used in medical procedures; a user interface for acceptinginputs from the user for selecting one of the user tool models; a usertool generator using one or more of the computers for executing softwarefor generating a realistic tool image of the selected user tool modelfor displaying on the display; and a user interface that can track themotions of an actual surgical instrument being used by the user with theparticular patient, such that the motions are used for dynamicallymanipulating the selected user tool image and/or the image of thetissues for dynamically interacting with the realistic image of thetissues during the simulated medical procedure for display to the useron the display in real-time.

Further provided is a method of performing a surgical simulation,comprising the steps of:

-   -   providing a computer system;    -   providing a display connected to the computer device;    -   obtaining patient image information about the biological tissues        of a particular patient for storing in the computer system;    -   generating, using the computer system, a dynamic image of the        biological tissues of the particular patient for display on the        display, the generating utilizing the patient image information        such that the dynamic image of tissues is displayed on the        display realistically representing corresponding actual tissues        of the particular patient;    -   generating, using the computer system, a user tool model for        dynamically interacting with the dynamic image of tissues via        manipulations input by a user for display on the display;    -   adjusting, using a user input to the computer system, the        dynamic image of tissues displayed on the display by adding or        modifying features of the tissues for display to compensate for        anatomical structures that are in the actual biological tissue        of the particular patient but are missing from the dynamic image        of tissues originally displayed such that the dynamic image of        tissues displayed are subsequently displayed on the display with        the added or modified features; and    -   generating, using the computer system, a realistic simulation of        the medical procedure for display on the display showing        interactions between the dynamic image of tissues and the user        tool model according to inputs by the user.

Also provided are additional example embodiments, some, but not all ofwhich, are described hereinbelow in more detail

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the examples of the present inventiondescribed herein will become apparent to those skilled in the art towhich the present invention relates upon reading the followingdescription, with reference to the accompanying drawings, in which:

FIG. 1 provides a high-level schematic of an example Surgical Theatersystem;

FIG. 1A provides an example computer system structure for implementingan example Surgical Theater System;

FIG. 2 a high-level diagram of an example of the Collaborative Theaterconcept using a plurality of the Surgical Theaters networked together;

FIG. 3 shows an example breakdown of a distributed simulation networkconcept for example Surgical Theater embodiments;

FIG. 4 is a block diagram showing example software functionality for anexample Surgical Theater system;

FIG. 5 is a diagram showing high-level Realistic Image Generator (RIG)platform;

FIG. 6 provides an example high-level architecture and workflow of aSurgery Rehearsal Platform (SRP) for an example Surgical Theater system;

FIG. 7 provides an example computer architecture for the example SRP;

FIG. 8 is a flow chart showing example tools for adjust the dynamictissue images;

FIG. 9 is a screen shot showing example interactive tool and tissueelements;

FIG. 10A and 10B are images showing example hands-free inputinteractions;

FIG. 11 is a screen shot showing an example marker and tools that can bedragged by a user;

FIGS. 12A, 12B, 13A, and 13B are screen shots showing example structuresthat can be modified using a magic tissue wand tool; and

FIGS. 14A and 14B are screen shots showing example structures that canbe modified using tissue painting tool.

DETAILED DESCRIPTION OF THE EXAMPLE EMBODIMENTS

FIG. 1 provides an example embodiment for one application of the system1 where a patient specific scan image (CT, MRI or similar) (14) is fedto the system's console (10), an algorithm that creates a 3 dimensionalrealistic anatomy display (18) adds texture, shadow, shadowing and othercues to the image, a mechanical properties algorithm (16) assignsmechanical behavior characteristics to the image and transfer the imagefrom static/still image to a dynamic and interactive image/model.Interfaces with or without force feedback (20) are connected to thesystem allowing the surgeon/operator (12) to manipulate the image/modelthat the system creates; the surgeon can select tools and implants fromlibraries of tools and implants including characteristics of those toolsand implants. The surgeon then performs a virtual surgery on amanipulateable, dynamic and interactive 3 dimensional image/model of hispatient organism in a realistic and dynamic manner.

The system includes an executive program that runs and manages all thesystem components and updates the status of the sub components accordingto the surgeon/operator (12) actions. For example, when the surgeon usesthe interface (20) to push a tissue (such as by using a chose tool) thathe sees in the display (18), the mechanical properties model (16)receives the information regarding the force that was applied, e.g., thedirection of force; the tool that is being used including its materialand shape and other mechanical characteristics of the tool, then themechanical properties are used to calculate a new state of the 3dimensional orientation an ad setup of the image according the forcethat was applied, the executive program send the calculated 3dimensional matrix to the realistic anatomy display (18) that wascreated by the mechanical properties algorithm (16), the realisticanatomy display calculates the new image and its cues due to the changeof image e.g., a new set of shadows and shadowing due to the neworientation of the image components are determined. Simultaneously, themechanical properties model (16) send a set of parameters to the forcefeedback interface (20), these parameters include information of theforce that the surgeon/operator (12) needs to sense due to theinteraction with the organs (the force that the organ returns after thesurgeon pushes or otherwise interacts with the tissues). This process ofcalculation of new stage at each one of the system's components (14, 16,18, 20) is executed rapidly and continuously in cyclic manner, and eachcycle is completed within a frame time of milliseconds, allowing thesurgeon/operator to receive real-time and realistic cues and real-timereactions to his actions.

The Surgical Theater is a system, as shown in FIG. 1A, that integratesone or more computers (PCs) 2A-2 n, one or more databases 3A-3 n andother hardware components (e.g., networks 5, 6) and proprietary softwareinto one complete system 1 (see both FIGS. 1 and 1A) that is structuredinto an immersive chamber/console sized about as big as a small walk incloset (see console 10 in FIG. 1). Once the surgeon 12 starts thesystem, the surgeon loads the set-up parameters of his patients whichinclude details of the patient to allow the system to up-load therelevant data, the Surgical Theater than loads all the patient'savailable CT and MRI images from a patient images 14 into thedatabase(s) 3 and other information that concern the simulated modelssuch as patient age, gender, and so on (some or all of which may beobtained from external entities 8, such as medical databases, forexample). The system utilizes tissue information parameters 16 from asystem database. The system 1 performs a segmentation process andidentified the Entities of the organ, Entities are vessels, tissue,tumor, and so on to create the simulated image model 18 shown to thesurgeon on the display of the device. The system provides realistictactical feedback 20 via feedback mechanisms to add further realism tothe simulation.

The system applies the layers of the realistic visual, the mechanicalproperties and other relevant parameters 16 from the system database(s)and characteristics relevant to the case, all applied on the top of theCT and MRI images 14 from the patient images database(s) 3 andsynchronized with those images. The synchronization creates, forexample, vessel mechanical properties that are ‘clamped’ or ‘attached’to the vessel images and so on to provide realistic simulationcapability. The surgeon can be provided the ability to “fine tune” themodels and adjust the mechanical properties of a certain area of theorgan. For example, the surgeon may adjust the elasticity and othermechanical characteristics of the Entities behavior.

Subsequently, after such a set-up, the Surgical Theater projects the 3dimensional organ model 18 presented in a realistic visual fidelity withrealistic features such as; texture, shadowing and other features thatadds realism to the simulated image. Each segment of the visual model 18is coordinated and corresponds with an appropriate mechanical propertiesmodel from the system database 16 and other relevant properties of thespecific case.

At this stage, the system allows the surgeon to browse and chooses fromthe system's virtual libraries 16 in the system database the relevantsurgery tools and other elements (in the system software terms thosetools and elements are “Entities” as well) that he may need to performthe surgery (or other procedure). Such elements may include; seizers andclamps, clips for aneurysm, artificial heart valves, and other elementsappropriate for the specific case. (Adding additional systems 1′, 1″ . .. connected to the system 1 via a network 9—such as over the Internet ora private network—can result in a collaborative theater platform,described in more detail later in this disclosure.)

All of the various Entities are represented by the system inhigh-fidelity distributed models and functioning in a distributedarchitecture, e.g., each Entity typically has a separate subEntity,where the subEntity is, for example, a “visual entity” or “mechanicalentity” and so on. Each subEntity exists in one of the differentenvironments (e.g., the visual system environment, the mechanicalmodeling environment and so on, described in more detail below)distributed among a plurality of computers. Each such subEntity isresponsible for its own performance (e.g., presenting the realisticvisual of the Entity, or performing the Entity's mechanical operation).

The subEntities communicate via a distributed network (described in moredetail below) to synchronize and coordinate the subEntities into a oneintegrated Entity compound model. For example, when a tissue is beingpressed by a surgery tool, the surgery tool pressure characteristics(e.g., the location, orientation and amount of pressure and so on) isdistributed via the network, each one of the subEntities is responsiblefor ‘listening’ and concluding if it is being affected by this surgerytoll pressure; once a subEntity determines that it is being affected,each such subEntity (for example, tissue Entity) models the affect ontheir subEntity model, e.g., the visual subEntity, presents the visualeffects (such as bloodiness of the tissue), and the mechanicalproperties models the shift of the tissue. Each subEntity distributesthe change—for example, the tissue location and dimension changes—overthe network so the other subEntities will be able to determine if theyare being affected by this change. At the end of such action, all thesubEntities of the tissue for the above example, (and the otherEntities), become accustomed to, and, if needed, adapt their states andthe models to, the new action that was sourced and initiated, in theabove example, by the surgery tool.

Thus, the various functions (subEntities) can be distributed amongvarious computers connected in a peer-to-peer network utilizingdistributed data and state duplication (for keeping local copies of thestate of the simulation), all listening on the network for any actionthat impacts their portion of the simulation, in which case they updatetheir parameters via the network to keep the system accurate, which may,of course, impact other functions in other subEntities, which willtherefore catch that fact by their monitoring of the network, leading tofurther updates, and so on. In this way, the system distributes thefunctionality among many computers in a parallel fashion so thatupdating can occur much quicker than it could if only a single computerwere used. Only those subEntities impacted by a change need respond, andthus network traffic can be reduced to essentials.

The Surgical Theater allows the surgeon to record his actions and savethem for later playback, to demonstrate his surgery plan to the chiefsurgeon or resident, or, to share information with other surgeons,demonstrate new techniques he is working on, practice the surgery, andso on. The system's interfaces to the surgeon includes surgeryinterfaces (e.g., seizers handles) that include force feedback that isdelivered to those tools to allow the surgeon to sense the forcefeedback cue of his actions, realistically simulating an actualprocedure.

Once the surgery tools and the other Entities are selected by thesurgeon, they are integrated into the virtual surgery scene and turninto an integrated element of the simulated scenario including realisticvisuals features and mechanical properties and operation propertiesfeatures that are applied to each one of those selected items. Forexample, the simulated scissors reflect mechanical characteristics ofreal scissors and will cut in the simulation as the real scissors do,and, aneurysm clips, when placed at the simulated vessel, simulatesblocking the blood flow.

Next, the surgeon performs the surgery actions at any stage of thevirtual surgery; the surgeon can “freeze” the simulation and rotate theorgan to observe the area of his interest from different orientationsand perspectives. The surgeon can “mark point of time” of the virtualsurgery and can command a “return to the mark point”. For example, thesurgeon can mark the time before clamping an aneurysm and return to thispoint of time while “un-doing” all the actions that took place afterthis point of time. In this fashion, the surgeon can evaluate differentsurgery approaches of a selected phase of the surgery without restartingthe entire surgery from the original starting point. Several such ‘markpoints’ are available allowing the surgeon to return and “re-do” actionsand exams/rehearse on several selected phases of the surgery. SurgicalTheater use may include surgeon rehearsals toward a surgery; surgeondemonstration to the chief surgeon or resident; surgical practice anddevelopment, testing, and validation of tools and methods, and knowledgesharing. Hands-free operation, as described below, can be utilized forthis feature.

Collaborative Theater

FIG. 2 shows a high-level example implementation of the CollaborativeTheater concept that was introduced with the Surgical Theater. Byleveraging next generation broadband infrastructure 25, individualsusing SRPs 21, 22, 23 . . . from different hospitals will be connectedallowing surgeons across the nation and across the globe tocollaboratively plan a surgery case, e.g., surgeons from two or moredistributed sites step into their SRP and rehearse, together, on apatient case toward a surgery. This Collaborative Theater allowssurgeons to study the best practice methods by observing previousSurgical Theater cases as well as providing remote teaching andmentoring. The Collaborative Theater allows all the hospitals that areconnected and using the SRP to gain access to the up to date accruedknowledge and most recent “best practices”. Again, hands-free operation,as described below, can be used for the collaborative theater concept.

System Level Design

The system level design description is outlined in the precedingsections. The visual rendering engines analyze 3D MRI and CTpatient-specific images and create computerized segmented modules thatrepresents the anatomical structures and features of the particularimage. The medical market has a vast number of advanced Digital Imagingand Communication in Medicine-DICOM (1) viewers. Their feature setsrange from layered black and white slices in 3 different panels thatcould be cross-referenced to a complete ability to fly through staticsubsets of 3D images of patient's organs. In addition, there are 4D and5D features that record various functional and dynamic changes of organsin a form of a movie clip. As magnificent as those captured images ormoving sequences might be, they are a fixed set of snapshots images intime.

The Surgical Theater takes existing 3D conversion processes and adds thefeatures specific to the human tissues and structures based on physicaland mechanical properties that are then stored in the system database.Once this patient-based model is set in motion in the virtual world, theSurgical Theater introduces a set of virtual surgical tools that allowthe surgeon to manipulate (push, cut, clamp, etc.) those models similarto real surgery tissue manipulation, providing an intuitive experiencefor the surgeon.

FIG. 3 provides a breakdown of an example Surgical Theater distributedsimulation network (Surgical Theater DIS (ST-DIS) is presented). Each ofthe components (i.e., blocks) in the figure is an isolated computationstation (that can be executed on a stand-alone computer or collection ofcomputers) with a designated set of functions. The stations areappropriately connected with a regular support network 31 (such as anEthernet network, for example) that handles slow irregular traffic, liketransferring of vast amounts of DICOM data. Upon more intense dataprocessing demand, the stations are supported by a specializedDistributed Interactive Simulation (ST-DIS) Network 32 that is ahardware isolated network used only for high priority simulation data(which can be implemented in high-bandwidth Ethernet, for example). TheST-DIS Network 32 carries volatile simulation information and allows forsuch an exquisite simulation load distribution.

The Surgical Theater's ST-DIS is a network architecture for buildinglarge-scale virtual worlds from a set of independent simulator nodes.The simulator nodes 33-38 are linked by the networks and communicate viaa common network protocol (such as TCP/IP, for example). The ST-DISinfrastructure enables various simulators to interoperate in a time andspace coherent environment. In the Surgical Theater's ST-DIS ST-DISsystem, the virtual world is modeled as a set of “Entities” thatinteract with each other by means of events that they cause. Thesimulator nodes 33-38 each independently simulate the activities of oneor more of the Entities in the virtual world of the simulation andreport their attributes and actions of interest to other simulator nodesvia messages on the network. The other simulator nodes on the networkare responsible for “listening” to the network messages, determiningwhich ones are of interest to them (based on the Entities they aresimulating) and responding appropriately.

One of the features of the ST-DIS network and simulation architectureconcerning distributed interactive simulation is that there need be nocentral server or processor. Each simulation application maintains itsown copy of a common virtual environment in its own memory or database.Representations of this environment are distributed by various means toall simulation applications prior to any real time operation. ST-DIS isbasically a peer-to-peer architecture, in which data is transmittedavailable to all simulators where it can be rejected or accepteddepending on the receivers' needs. By eliminating a central serverthrough which all messages pass, ST-DIS reduces the time lag for asimulator to send important information to another simulator. This timelag, known as latency, can seriously reduce the realism, and thereforethe effectiveness, of a networked simulator. Effective distributedsimulation depends on very low latency between the time that a newstate/event occurs for a simulated entity to the time that thestate/event is perceived by another entity that must react to it. Anydelay introduced by the training device could result in negativereinforcement to the trainee.

Referring again to FIG. 3, the Archive Server 34 is generally used toperform the tasks of downloading and retaining in a database largeamounts of data necessary for simulation. In addition, the ArchiveServer 34 can be used to prepare obtained data for further use in thesimulation. Note that because its duties are typically global in nature,and not critical to the simulation activity, the Archive Server 34 istypically only connected to the support network 31.

FIG. 3 shows a network architecture that includes an off line “support”network (31) that “Archive Server” (34) that loads the medical images(CT/MRI) and additional initialization data stored in a database (forexample, the patient name, age and so on and files to be included in thescenarios such as surgery tools libraries) “Debrief Server” (38) thatrecords control inputs and store the scenarios and all the actions in atimeline information and allows playback of scenarios and actions. Thereal time network (32) is the network that transfers messages betweenthe systems node during the simulation in a real time fusion—one way forimplementing this network can be a Distributed Interactive Simulation(DIS) network (32), the components that connected to this network are;Control Input (33) that connected to the surgeon/operator systemsinterfaces, this node has an optional direct physical connection to theHost Computer (35) that may be implemented in a case that the real timerequirements of the system cannot be satisfied by the DSI network and adirect physical connection between those node sis needed. The HostComputer (35) includes the executive manger program and other models andsimulation components and it is responsible for the real timesynchronization and timing of the entire systems.

The Theaters Initialization Systems (TIS) (36) performs that systemallocation and setup for each one of the nodes, for example, when thesurgeon select a specific tool to use, the TIS allocates/activates theappropriate models of this tool for generating an accurate toolsimulation (with tool characteristics stored in a database) for all thenodes assuring that all the nodes are set up with the sameinitialization. The Image Generator (36) performs the rendering andvisualization tasks of the scenarios. The Host Computer (35), the TIS(36), the Image Generator (36) and the Debrief Server receive andexchange information with off line for initialization from the Supportnetwork (31) and receive and exchange information with the real timenetwork (32) for “on line” and real time simulation.

Needed organ surface and volume data are extracted from an existingMRI/CT scan stored in the database. To obtain 3D organ surface data, thesystem can use a DICOM viewer and data management system such as theOsiriX (or comparable) that is open source software implemented forApple Macintosh computers, for example. By “tapping into” OsiriX'sability to generate 3D surfaces of organs and organ groups based on thevoxel density values with Objective C source code, the Surgical Theateradds an ability to store information about the 3D surfaces and organtypes that describe into a flat file in a database. The entire set ofparts of this study stored in this manner in the system database so thatit is later transferred to the Image Generator Station 37 that recreatesthe patient-specific images based on standard characteristics of theorgans. Once the necessary rendering data is obtained, the renderingplatform for Image Generator Station 37 is applied to the image. Forthis, a proprietary Image Generator algorithm is integrated (such as aFlight IG; see the features in the separate headings for the RealisticImage Generator-RIG) with a Visualization Tool Kit.

The IG has unique features that deliver fine cues such as shadowing,texture, and material properties that are assigned to the visual modelsand as further detailed in the RIG sections. Not only does the IG createrealistic and fully immersed environments by using those features, itcan also process large volume of visual data base models under hard realtime constraints. Enabled by the combination of the DIS architecture andthe “Entity” design, the network traffic is minimized and the anatomy ofthe peer-to-peer nodes create a highly efficient real time system.

After the patient-specific images have been successfully rendered,various physics libraries are added in order to create propersimulation. Pushing and manipulation of the brain tissue is simulatedusing extensive research embodied in modeling platforms such as theOpenTissue (or comparable) collections of libraries that are available.The OpenTissue, for example, is an open source collection of librariesthat models volumetric shells and other complex behavior of3-dimensional shapes. Customized libraries can also be developed foruse. Specificity of the brain tissue physics and mechanics propertiesthat derived from the research of mechanical properties of brain tissuein tension can be utilized, for example. Experimental papers areavailable that provide mathematical models of the mechanicalmanipulation of animal brain samples. Dynamic and realistic interactionof simulated surgical tools with the simulated tissues are implementedin the algorithms and approaches as described in co-pending patentapplication PCT/US12/31514 filed on Mar. 30, 2012, and U.S. Pat. No.8,311,791 filed on Oct. 19, 2010 (incorporated herein by reference). Thework looks at various tools and tissue types to create a realisticsimulation specifically for implementation of surgical simulations.

The software code of the example Surgical Theater is written in acommercial environment such as C++, with the code being designed to runin windows operating system, a Linux system, or compatible. In thecoding development process, emphasis is given for the code real timeexecution and code efficiency all aimed to maintain a real time systemperformance while minimizing the latencies.

The visual system driver located in the Image Generator (37) is designedwith an optimizers environment, such as OpenGL or similar, enableshigh-performance rendering and interaction with large models whilemaintaining the high model fidelity demanded, providing attention todetail while maintaining high performance in a cross-platformenvironment.

For computing efficiency purposes, each of the visual model's Entitieshave several Level of Details (LOD) representations; high LOD ispresented in areas of the simulation scene in which the surgeon needshigh resolution at, and, lower LOD is presented in areas of thesimulation scene in which the surgeon has no immediate interest orinteraction with. For example, tissue visual model is presented in highLOD in the area around the surgeon interaction and with lower LOD inareas that the surgeon doesn't have immediate interaction with. The LODcan be dynamically adapted: a surgeon's actions such as pointing thesurgery instruments toward a specific area can be utilized by the LODoptimization algorithm for the dynamic allocation of the LOD forspecific section of the visual models.

The typical system's computer is a PC with a multiple core (multipleprocessors) which provides flexibility and growth potential. Thecomputer system includes random access memory, Ethernet ports, systemdisk, and data disk.

For the validation of the Surgical Theater (image quality, realism,image controller and manipulation), the skills and experience of seniorsurgeons are utilized. The surgeons are used to evaluate the system byperforming specific surgical procedure while comparing it against theirvast neurosurgical experience as well as against a specific case thatthey have already operated and is being simulated in the SurgicalTheater.

The Surgical Theater Block Diagram of FIG. 4 describes the functionalityand the flow of the process (vs. the actual network connection of FIG.3) from the row data of the scanted image DICOM 41 through the processof segmenting the row data (to identify soft tissue, vessels and so on).Then the Image Generator assign visual representation of each segment(shadow texture and so on), this image is connected via the DIA 44network to a projection interface 46 and to the Host 45 that will updatethe image generator 43 with the surgeon actions that are connectedthrough the Surgeon Interface 47 and the mechanical Properties and othermodeling that the Host includes that all will reflect the new state thatthe Host will send to the IG 43 during each simulation cycle.

By eliminating the central server through which all messages pass,ST-DIS dramatically reduces the time lag for one simulator (computer) tosend important information to another simulator (computer). This timelag, known as latency, can, if too large, seriously reduce the realism,and therefore the effectiveness, of a networked simulator. Effectivedistributed simulation depends on very low latency between the times anew state/event occurs for a simulated entity to the time thestate/event is perceived by another entity that must react to it. Anydelay introduced by the training device results in the negativereinforcement to the operator (e.g., the surgeon).

According to the recommended practice for communications architecture(IEEE 1278.2), the underlying communications structure should support100 ms or less latency for packet exchange for closely coupledinteractions between simulated entities in real-time (e.g. simulatinghigh performance aircraft in a dogfight or simulating a surgeonperforming brain surgery). This requirement is based on human reactiontimes that have been the basis of Human-In-The-Loop (HITL) flightsimulator designs for many years.

Within the ST-DIS system, the virtual world is modeled as a set ofEntities (as described previously) that interact with each other bymeans of events that they cause. An Entity is a sub-component in thesimulated scenario, such as tissue, specific characteristics (suchas—tissue mechanical properties,) creating a sub group of that “tissueentity”. Another Entity can be a blood vessel, for example, and so on.Each Entity can have several subEntities that operate in a distributedmanner (such as on different simulators/computers). Together, thosesubEntities are combined to create the complete Entity model. ThosesubEntities are, for example: the Visual subEntity that holds andsimulates the Entity's visual feature and characteristics, or, theMechanical Properties subEntity that holds and simulates the Entity'smechanical feature and characteristics. Each of those subEntities modelcode can run in a different computer (or group of computers) such as aPC, and they communicate with each other as well as with other Entitiesvia the ST-DIS network. The simulator nodes, independently simulate theactivities of one or more Entities (or subEntities) in the virtual worldof the simulation and report their attributes and actions of interest toother simulator nodes via messages on the ST-DIS network. The othersimulator nodes on the network are responsible for “listening” to thenetwork messages, determining which ones are of interest to them (basedon the entities they are simulating) and responding appropriately.

The above-described Surgical Theater architecture is based on thisDistributed Simulation concept thereby enabling pioneer and exclusiveabilities to deliver a premier fidelity which is an essentialrequirement for creating immersive scenarios crucial for the rehearsingof open/classic surgeries where the surgeon(s) interacts with theorgan(s) by direct human sense. As each Entity is divided to itssub-components (visual, mechanical properties and so on), and as each ofthose subcomponents/Entities' simulation code runs in a separatecomputer, this can maximize the computation power, and by that thecreation of a unique and exclusive premier fidelity, fine cues, andcomputing capabilities while handling terabytes of information underhard “real-time” constraints while maintaining real time performance(e.g., less than 100 millisecond latency), the core capability of theFlight Simulation technology.

The Surgical Theater facilitated a visual rendering engine whichanalyzes 3D MRI and CT patient-specific images and creates computerizedsegmented modules that represents anatomical structures and features ofthe particular image. Medical market has a vast number of advanced DICOMviewers, but as magnificent as those captured images or moving sequencesmight be, they are based on a fixed set of snapshots in time. TheSurgical Theater takes existing 3D model conversion algorithms and addsthe features specific of the human tissues and strictures based onphysical and mechanical properties creating a “living” image with modelsthat reforms the patient specific CT/MRI images according to actionstaken by the surgeon and based on the models that simulate themechanical properties of each pixels in the image and realistic visualcharacteristics models. Once this patient-based model is set in motionin the virtual world, a set of virtual surgical tools (that can includeaneurysm clips and clip appliers, implants such as bone joint implants,or other devices) are introduced allowing the surgeon to manipulate(push, cut and etc.) those models similar to a real surgery tissuemanipulation. Thus, the Surgical Theater provides an intuitiveexperience for the user.

For the Image Generator, the Surgical Theater of the example embodimentintegrates a proprietary Flight Simulation Image Generator algorithmwith a visualization code such as Visualization Tool Kit (VTK). Asdetailed in the following sections, the Surgical Theater Realistic ImageGenerator has features that deliver fine cues such as shadowing,texture, and material properties that are assigned to the visual models.

The Realistic Visual Sub System

This section focuses on the “realistic visual” segment of the SurgicalTheater that is a modification of a Flight Simulation Image Generatorthat is capable of rendering satellite images into realistic 3dimensional images and models that are converted into the SurgicalTheater realistic Image Generator (MG) handling and real time renderingCT/MRI DICOM images into a patients' specific realistic and dynamicCT/MRI images and models that are crucial for the open/classic surgerieswhere the surgeons interact with the organ by direct human sense.

The use of a visual system in the creation of the immersive simulationsystem in the field of Human factor Engineering is important; studiesdemonstrate that a high percentage of the immersion is constructed andcontributed by the level of fidelity and realism of the visual systemthat the operator (e.g., pilot or surgeon) interacts with. Findings showthat operators who rehearse on high fidelity visual systems completedthe memory task including self-report of confidence and awareness statesin significantly higher levels than the low fidelity group. Asignificant positive correlation between correct ‘remember’ and ‘know’responses, and in confidence scores, are found when utilizing highfidelity, realistic simulation.

As outlined above, the Surgical Theater creates a realistic “life-like”digital rendition of the surgical site and the surroundingtissues/structures. Since this digital rendition is patient-specific and“life-like”, it sets Surgical Theater apart from other simulators thatuse generic imagery to create approximate renditions of the surgicalsite, or, other system that simulates noninvasive procedures such asendoscopic, vascular and similar procedures, where the surgeon/operatorinterfaces the organism with a camera that has its own visualcharacteristics that are defined and limited by the camera specificationand are very different from the visual characteristics of the bare anddirect eyes view of the open/classic surgeon's where the surgeoninteracts with the organism with direct sense of his eyes However,realistic “life-like” rendering presents a surmountable task due to thecomplexity of the properties of the living biological tissues. In orderto create such high degree of realism, the Surgical Theater includes aReal Image Generator add-on (RIG): a visual system wherepatient-specific images of the surgical site, together with surroundingtissues, is realistically presented and can be manipulated in thisall-purpose manner.

FIG. 5 shows a RIG Architecture Block Diagram. Data Base box—collectionof the mesh modules based on the patient-specific CT/MRI, 3D andsegmented images, pre-processing of the images, smoothing, masking,scaling. Graphic Creator box—Interface to the graphics card. ST-DISInterface box—Interface to the ST-DIS network. The figure shows ahierarchy diagram of the visual systems. The system includes anexecutive program that runs and manages all the system components andupdates the statutes of the sub components according to thesurgeon/operator and the status of all the sub components as they areread through the DIS network (502). The Operating/Executive Engine (501)is responsible for the initialization of all the software and hardwarecomponents in a way that all the system's components are working withthe same data bases (for example, the set of tolls that the surgeonchoose). When the scenario starts, the Operating/Executive Engine (502)performs the cycle and timing control and perform the task of managingeach component to complete its calculation cycle within the time framethat it is planned on in a way that all the system's sub componentsreceive the information from the other sub components on a timely mannerallowing the overall system to complete the simulation cycle in a giventime frame. For example, when an action is taken by the surgeon andtransmitted by the DIS network (502), the Feature Generator (504) readsthe relevant part of this action/consequence of this action ascalculated by the mechanical properties algorithm, the Graphic Creator(503) change the image according to this action (for example, move avessels that was pushed by the surgeon), then calculates the changesthat need to be applied on the image as a result of this change, forexample, creating a shadow resulted by the change of the vessel locationand orientation. This cycle is executed rapidly and continuously managedby the Operating/Executive Engine (501) in a cyclic manner in a way thateach cycle is completed within a frame time of milliseconds allowing thesurgeon/operator to receive real time and realistic cues.

SRP General Description:

The SRP creates realistic “life-like” full immersion experience for theneurosurgeon to plan and physically rehearse cerebral aneurysm clippingsurgery by converting patient-specific DICOM data of the surgical siteand surrounding tissues/structures into a dynamic and interactive 3Dmodel. Unlike existing surgery preparation devices, the SRP can provide:(i) fine cues of look, feel and mechanical behavior of patient-specifictissues, (ii) 3D display of the patient-specific anatomy, (iii)real-time, surgery-like manipulation of 3D tissue model and, in thefuture, (iv) haptic feedback to the surgeon for a “full immersion”experience. Due to the complexity of organization and mechanicalproperties of living biological tissues, developing such a realistic“life-like” rendition will require following sub-developments (FIG. 6):(i) DICOM Image Volume Reader (602) and Viewer with built-in segmentedVolume of Interest (VOI) Model Generator (611), (ii) 3D Image Generator(IG) (604), (iii) Real Time Soft Tissue Deformation Engine (RTTDE)(612), (iv) Surgical Distributed Interactive Simulation (SDIS) Network(610) (v) Simulation Executive Application (SimExec) software (601) (vi)Surgeon User Interface (SUI) (605), and (vii) User Interface InterpreterEngine (UIIE) (613) (vi) VisChasDB database for the visual such as toolslibrary heartbeat, blood flow and others (603).

The conversion of a set of 2D patient-specific DICOM data into asegmented 3D VOI Model with accurate patient-specific tissue attributesis done using DICOM Volume Viewer (611) (proprietary software developedby Surgical Theater LLC). First, patient-specific DICOM data setundergoes image enhancement stage using mathematical algorithms adaptedfor a 3D dataset (603). This enhancement stage will increase imagesmoothness and reduce image noise without affecting the ability todistinguish between different tissue types.

Next, using a multi-panel view window within the DICOM Volume Viewer(602), the surgeon defines VOI, i.e. surgical site containing aneurysmand surrounding vessels and structures. The next step is tissuesegmentation, i.e. initial tissue-specific intensity ranges are assignedto tissues using Top View window of DICOM Volume Viewer to yield 3D VOIModel with high-resolution, quality, customizable data structure, andtissue-specific segmentation. The 3D VOI model is stored in apatient-specific repository and accessed during the cycle of operationas follows: (I) 3D Image Generator (IG) (604) presents the surgeon withhigh-fidelity visual representation of the model via graphicalinterface; (II) the surgeon manipulates the model using realisticsurgical tools inside the Surgical User Interface (SUI) (605); (III)User Interface Interpreter Engine (UIIE) (613) translates surgeon'smanipulations into a set of mathematical values that together with otherpatient-specific inputs (e.g. heartbeat, blood flow and others) areapplied to the model by the Real Time Tissue Deformation Engine (RTTDE)(612). As the model changes, the IG (604) reflects those changes to thesurgeon in real-time, thus completing one simulation cycle. Smooth,continuous, “life like” SRP flow is achieved by repeating cycle ≥60times per second by the IG and 20 times per second by the RTTDE (612).

SDIS based architecture:

The SDIS based architecture facilitates a unique and exclusive abilityfor premier fidelity, fine cues and computing capabilities whilehandling large volume of information under hard real-time constraintswhile maintaining real time performance which is the core capability ofthe Flight Simulation technology. One of the features of the SDISnetwork is that there is no central server or processor, each simulationnode (nodes may be: Image Generator, User Interface, Mechanical Modelingcomputer and so on) maintains its own copy of the common virtualenvironment—vessels, tissues and other models that are held andmaintained at each of the simulation node; each such model is handles asa separate “Entity”. This architecture enables several PCs to worktogether in a synchronized manner under hard real time constraintsallowing SRP's pioneering and unique capabilities to deliver a premierfidelity of the simulated scene. This creates an immersive scenario thatallows rehearsal of open/classic surgeries where the surgeons interactwith the organ by direct human sense.

Once the surgery tools and the other Entities are selected by thesurgeon, they are integrated into the virtual surgery scene and turninto an integrated element of the simulated scenario including realisticvisuals features and mechanical properties and operation propertiesfeatures that are applied to each one of those selected items, forexample—the scissors have the real mechanical characteristics and willcut as the real scissors do, and, Aneurysm clips, when placed at thevessel, blocks the blood flow.

The SRP system as is compose by the following units or combination ofsub parts of the units depended on the configuration, volume that needsto be simulated and the specific application. These are similar to thosefor the Surgical Theater system as shown in FIG. 4, but modified asdescribed in this section. The sup components can run in Severalseparated Computing Processor Units in multiple PCs (FIG. 7):

The workstation that the surgeon works on is the User Interface 101. TheImage Generator 102 operates similarly to the like device in theSurgical Theater. The Simulation Executive Manager 103—synchronizes thereal time operation of the system, runs, and executes the modelingprograms. The STDE Workstation 104—This PC handles the STDE (Soft TissueDeformation Engine). The Archive Server 105—This station holds all therelevant files and data and able to record the procedure for futuredebriefing and data collection, and this PC also serves as the networkdomain controller. The IOS (Instructor Operation Station) 106 is formonitoring and controlling the training session, also allowing theinstructor to “inject” events. Also serve as the “Master of Ceremony”and will activate the whole training session. One or more User Interface107 is provided for hand-free control and/or for tracking real surgicalinstruments, as described below.

Each of these Computing Processor Units connects via the SDIS networkwith a network switch (not shown).

Surgical Interface

As discussed above and in the related applications, the updated SurgicalTheater provides a method for a hybrid rendering (volume and surface) ofimages from a scene file (for example, a medical scan file) of multipleformats (for example, a Digital Imaging and Communications inMedicine—DICOM) into an interactive image/scene. The output image/scenemay be 2-dimensional or 3-dimensional and will contain geometry,viewpoint, texture, lighting, shadow and shading information and otherelements of the description of the virtual scene. FIG. 8 shows a flowchart showing the updated features, with the specific details discussedhereinbelow.

The interactive image/scene is constructed from elements that are bothvolumetric rendered elements and surface rendered elements. Furthermore,each element, volume or surface, interacts with one or more elementsthat are volume (see 112 of the image shown in FIG. 9) and/or surfaceelements (see 111 of FIG. 9).

Interaction between elements includes, but is not limited to, physicalinteraction such as: a collision model implemented to represent theinteraction between elements that results with movements and/or reshapeof elements that replicate the actual physical movements of the elementaccording to physical conditions, such as pressure, elements material(elasticity, stickiness etc.), and collision condition such as collisionangels and elements orientation.

The rendering process equation accounts for all lighting shadow adshadowing phenomena and produce a final output stream that incorporatesall the visual elements.

Surgical theater rendering software solves the rendering equation inreal time while reflecting the physical interaction between elementswhile maintaining the realistic look of output image/scene/model.

For example, in FIG. 9 a clip 112 presses a blood vessel (volumerendered element) that results a reshape of the vessels 111. Userscontrol the interaction control by either a mouse controller, a touchscreen, 3D or 6D controllers, or by a hands free controller, describedbelow.

Hands free controller or touch screen: by integrating a camera-baseddevice that captures and recognizes the users body's elements in realtime (in a manner that may utilized technologies similar to the Kinectsystem by Microsoft, for example—seewww.xbox.com/en-US/kinect/kinect-effect, with the Leap technology byLeap Motion being another example, see live.leapmotion.com/about.htmlboth incorporated by reference, see item 107 of FIG. 7), or by a touchscreen or any other interface, the user can interface and interact withthe image/scene by waiving with his hands in a pre-defined ways, tocontrol the image/scene (FIGS. 10A and 10B). The user can, among others,do the actions of:

Rotate, move, and shift the image/scene (see the hand motion shown inFIG. 10A to FIG. 10B, with the motion moving and re-orienting the image121, 122, respectively)

Zoom in and out.

Select elements from a library and add them to the image/scene.

Drag and drop elements from in the image/scene.

Command one or more elements to interact with one or more otherelements—for example, place a an aneurysm clip and command it to beclosed on the aneurysm and then command “close” which causes the clip(surface element) to interact with the aneurysm (volume element) withthe resulting physical squeezes of the aneurysm and the movement of theclip (form open blades to closed blades).

Select elements and remove them from the image/scene.

Scroll between slices if the image/scene is stacked/built from multipleslices (such as CT MRI)

Reposition objects in the scene by selecting them and then dragging themto the desired 3D position. This allows, but not limited to, cause toolto tissue interaction (in the case of a tool being dragged 132 in FIG.11) or to perform measurements in the scene (in the case of draggingmeasurement markers 131 see FIG. 11).

“Painted tissue”:

General: medical images produced from scanner (such as MRI, CT andothers) provide a physical, functional structural or other informationabout the scanned anatomical structure. Due to a variety of reasons,among others, the scanner limitation, not all the anatomical structuresare clearly visible in the resulted image. Two examples for thisphenomena/limitation are:

-   -   1—In MR scan nerves may not always be visible. Specific example        may be in images of brain cerebral scans toward treatment of        microvascular compression where a cerebral vessel touches a        nerve and creates a physical pressures on the nerve—in those        scans the vessel is often visible at the scanned image, yet, the        nerve cannot be observed.    -   2—In MR, CT, or other scans, a part of anatomical structure may        be visible, yet, due to verity reasons, among others, the        scanner limitation, only part of the anatomical structures is        visible. One example may be in images: in a CT or MRI scan,        parts of the vessels structure may be visible while other parts        are not. In this example, the vessel image will be distorted        and/or not completed.    -   “Tissue Painting”—the developed algorithm and software tool        provides the user an interface to draw any geometric shape or        free hand drawing shape in 2- or 3-dimensions (e.g., line,        circle, clinic, ball etc.). The resulted painting interface        allows the user to determine the thickness of the line or the        wall shell and walls of 3-dimensional shapes. The user can also        determine the visual characteristics of the shape; the color,        the transparency etc. The new shape is created in within the        medical image in a way that allows the new created shape to        become a part of the scan (Magic Tissue) and to be integrated        with the scan image. For example, the user can draw a line that        will represent a nerve. This nerve can be crated at a lengths,        shape, color, transparency, location and orientation of the user        selection. The user can place the shape in proximate to an        existing anatomical structure observed in the scanned image        (e.g., a visible part of the scan) and to “connect” it to an        existing anatomical structure. The user also assigns this newly        drawn shape to a specific tissue type. Once created, this new        shape is rendered and added to the 3 dimensional anatomical        model. The new shape can be rendered and reconstructed as a        volume model or as a mash/polygon model. FIGS. 14A and 14B show        examples of tissue painting at 181, 182, respectively.    -   “Magic Tissue Wand”—Due to a variety of reasons, including,        among others, the scanner limitation, not all the anatomical        structures are visible in the resulted image. Often an        anatomical structure (e.g., a blood vessel) will appear only        partially in an image and the entire structure will not be        visualized; there will be missing parts of the anatomical        structure and the anatomical structure will not be a whole        continued/completed one. An algorithm and software tool is        provided that completes the anatomical structure and        reconstructs the image to create a more complete anatomical        structure. The reconstruction algorithm utilizes analytical        geometric calculations and calculations performed on the scanned        image to analyses and to recreate the anatomical structure based        on existing ‘hints’, cues and other signs in the scanned image        in order to complete missing parts of the anatomical structure.        This includes geometric and spherical distributions of similar        voxel in the Hounsfield unit (HU) and the creation of vector of        distribution to analyze and recreate the missing part of the        anatomical structure (HU scale is a linear transformation of the        original linear attenuation coefficient measurement into one in        which the radiodensity of distilled water at standard pressure        and temperature (STP) is defined as zero Hounsfield units        (HU)—for example, the radiodensity of air at STP is defined as        −1000 HU).    -   The Magic Tissue Wand algorithm connects the spherical        distributions voxel in in a complimentary way—that is, voxel        will be added to the original, incomplete anatomical structure        (see item 152 of FIG. 12B and item 162 of FIG. 13A) if by adding        those voxel together, the anatomical structure is more complete.        (e.g., continues, combined into a whole/complete anatomical        structure, see item 151 of FIG. 12B5 and item 161 of FIG. 13B).        By applying the Magic Tissue Wand algorithm on to the scanned        image, anatomical structure will be completed. For example,        after the Tissue Wand algorithm has been applied, a vessel that        was not visible in a certain part of the image, will be        completed and will appear as a more continuous anatomical        structure (e.g., item 161 of FIG. 13B).    -   Volume and or mash/polygon reconstruction—the anatomical        structures that were created both with the Tissue Paint and        Magic Tissue Wand algorithm and integrated with the scanned        image are, for any practice consideration, an integrated part of        the image. For example, the vessel that anatomical structures        that originally was partial and complete, after applying the        Magic Tissue Paint and Tissue Wand algorithm will become a        complete anatomical structures with structure that is combined        from the original scanned image and the new created structure.        Furthermore, a control (check box) allows to select the new        created structure and to switch between on (showing the new        created structure) or off (hiding the new created structure).        Additionally, an option is provided for selection to render the        new created structure in a volume and or mash/polygon        rendering/reconstruction.    -   Marked Region—A developed algorithm and software tool provides        the user an interface to draw any geometric shape or free hand        drawing shape in 2- or 3-dimensions (e.g., line, circle, clinic,        ball etc.). The region that is included/enclosed/captured within        the said geometric shape (2- or 3-dimensions) is defined as a        “Marked Region”. The user then, has the ability to define and        assign any visual characteristics and any mechanical properties        to that “marked region”.    -   Visual characteristics; color/transparency/shading—the new        created structure either or with the Magic Tissue Paint, Tissue        Wand algorithm or the Marked Region can be presented in any        selected visual characteristics of color that can be selected        from a library of available colors, and a transparency that can        be selected on any level from 0 to 100. Furthermore, the        characteristics of shading and shadowing of the new created        structure can be modified by tuning the characteristics of the        virtual light sources. The virtual light sources characteristics        includes: spherical location in space, color of the light,        strength of the light, the aspect ratio, the geometric shape of        the virtual source etc.    -   Mechanical properties—the new created structure either or with        the Tissue Paint, Magic Tissue Wand algorithm or the Marked        Region can be assigned with mechanical properties        characteristics. That is, that a mechanical model of a specific        tissue can be coupled to the new created structure and        therefore, the new created structure will inherent such        mechanical properties and will react, dynamically and statically        according to those mechanical properties. For example, if a        “soft tissue” mechanical properties where assigned to a new        created structure, it will react according to a soft tissue. For        example, when it will be pushed by a virtual surgery instrument,        it will squeeze and reshape according to the force applied and        the tissue mechanical model. Furthermore, interaction between        new crated structures and other new crated structures,        interaction between originally scanned structures and new crated        structures and between new crated structures and originally        scanned structures are seamless. The mechanical properties        coefficients of any anatomical structure (stiffness, elasticity        etc.) can be tuned by the user to create a tailored made        mechanical behavior.    -   Real Time Tracking and Feedback—a system to track a real surgery        instrument during the surgery. The tracking system transfers the        surgery instruments location and coordination in space relative        to the orientation and location of a real anatomical structure        (for example, specific spot on the patient's head). The        instruments' location and orientation is then sent to the        surgical simulating system. Feedback is provided to the surgeon        based on the patient specific simulation and the instruments'        location and orientation. One example for such feedback can be;        the system generates feedback to the surgeons for the type of        tissue he is dissecting and alarming the surgeon in case that he        dissects healthy brain tissue instead of a tumor. Additional        example is that after that the surgeon applied an implement on        the real anatomical structure (for example an aneurysm clip        applied on an aneurysm on the real patient), the system allows        the surgeon to rotate the simulated image/model that is princely        oriented as the real anatomical structure based on the tracking,        and observe and evaluate the location and efficacy of the placed        implant.    -   This tracking and feedback of the real instrument can be        accomplished in a number of ways, such as by using a video        system to track the location and movement of the instrument and        the patient features. Alternatively (or in addition to video        tracking) the surgical instrument may be modified to enable        tracking, such as by using GPS, accelerometers, magnetic        detection, or other location and motion detecting devices and        methods. Such modified instruments may communicate with the        Surgical Theater using WiFi, Bluetooth, MICS, or other        communications methods, for example. Interface 107 in FIG. 7        can, for example, be utilized for this feature.    -   Many other example embodiments of the invention can be provided        through various combinations of the above described features.        Although the invention has been described hereinabove using        specific examples and embodiments, it will be understood by        those skilled in the art that various alternatives may be used        and equivalents may be substituted for elements and/or steps        described herein, without necessarily deviating from the        intended scope of the invention. Modifications may be necessary        to adapt the invention to a particular situation or to        particular needs without departing from the intended scope of        the invention. It is intended that the invention not be limited        to the particular implementations and embodiments described        herein, but that the claims be given their broadest reasonable        interpretation to cover all novel and non-obvious embodiments,        literal or equivalent, disclosed or not, covered thereby.

What is claimed is:
 1. A modeling system for enabling a user to performa simulated medical procedure, said system comprising: one or morecomputers; a display for displaying images to the user; a database forstoring characteristics of the tissues of a particular patient; an imagegenerator using one or more of said computers for executing software forgenerating a dynamic realistic image of the tissues of the particularpatient based on the stored characteristics of the particular patientfor displaying on said display, wherein said realistic image of thetissues is provided showing an appearance including shadowing andtextures indicative of the tissues of the particular patient; a databasefor storing a user tool library for providing a plurality of user toolmodels of actual user tools used in medical procedures; a user interfacefor accepting inputs from the user for selecting one of the user toolmodels; a user tool generator using one or more of said computers forexecuting software for generating a realistic tool image of the selecteduser tool model for displaying on said display; and a user interfaceincluding a camera for accepting inputs from the user based on motionsof the user's hands, said inputs for dynamically manipulating saidselected user tool image and/or the image of the tissues for dynamicallyinteracting with the realistic image of the tissues during the simulatedmedical procedure for display to the user on said display in real-time,wherein the dynamic interaction between the user tool image and theimage of the tissues is displayed on said display using images withrealistic visual features exhibiting realistic mechanical interactions.2. The modeling system of claim 1, further comprising a tool to adjustthe dynamic image of tissues to provide the ability to draw anygeometric shape on the dynamic image of tissues.
 3. The modeling systemof claim 2, wherein the tool to adjust the dynamic image of tissuesincludes a tool to provide the ability to complete an incompleteanatomical structure of the dynamic image of tissues.
 4. The modelingsystem of claim 1, wherein the tool to adjust the dynamic image oftissues provides the ability to modify the texture, lighting, shadowand/or shading of a portion of the dynamic image of tissues
 5. Themodeling system of claim 1, wherein the tool to adjust the dynamic imageof tissues includes a tool to provide the ability to command the tool tointeract with one or more portions of the dynamic image of tissues. 6.The modeling system of claim 1, wherein the tool to adjust the dynamicimage of tissues includes a tool to provide the ability to selectelements of a model of the tool and/or the dynamic image of tissues forremoval from the displayed image.
 7. The modeling system of claim 1,wherein the tool to adjust the dynamic image of tissues includes a toolto provide the ability to reposition objects in the displayed image byselecting the objects and dragging the objects to a desired position fordisplay in the image.
 8. The modeling system of claim 1, wherein thetool to adjust the dynamic image of tissues includes a tool to providethe ability to enhance and integrate anatomical structure in the dynamicimage.
 9. The modeling system of claim 1, wherein the tool to adjust thedynamic image of tissues includes a tool to provide the ability to adraw any geometric shape for adding to the dynamic image of tissues. 10.The modeling system of claim 1, further comprising: a database forstoring a library of a plurality of models of different implants; and auser interface for selecting one implant model from said plurality ofmodels for use with said user tool model for dynamically interactingwith said image of tissues.
 11. The modeling system of claim 1, whereina tool to adjust the dynamic image of tissues includes tools to providethe abilities to: draw any geometric shape on the dynamic image oftissues; complete an incomplete anatomical structure of the dynamicimage of tissues; or modify the texture, and/or lighting of a portion ofthe dynamic image of tissues.
 12. The modeling system of claim 1,wherein the inputs from the user based on motions of the user's handsare configured to track the motions of an actual surgical instrumentbeing used by the user with the particular patient, such that saidmotions are used for dynamically manipulating said selected user toolimage and/or the image of the tissues for dynamically interacting withthe realistic image of the tissues during the simulated medicalprocedure for display to the user on said display in real-time.
 13. Themodeling system of claim 12, wherein said user interface that can trackthe motions of an actual surgical instrument includes a GPS receiver, anaccelerometer, a magnetic detection device, or a camera.
 14. A modelingsystem for enabling a user to perform a simulated medical procedure,said system comprising: one or more computers; a display for displayingimages to the user; a database for storing physical characteristics ofthe tissues of a particular patient; an image generator using one ormore of said computers for executing software for generating a dynamicrealistic image of the tissues of the particular patient for displayingon said display, wherein said realistic image of the tissues is providedshowing an appearance including shadowing and textures indicative of thetissues of the particular patient; a database comprising a user toollibrary for providing a plurality of user tool models of actual usertools used in medical procedures; a user interface for accepting inputsfrom the user for selecting one of the user tool models; a user toolgenerator using one or more of said computers for executing software forgenerating a realistic tool image of the selected user tool model fordisplaying on said display; a user interface for accepting inputs fromthe user, said inputs for dynamically manipulating said selected usertool image for dynamically interacting with said realistic image of thetissues during the simulated medical procedure for display to the useron said display in real-time; a user interface including a camera foraccepting inputs from the user based on motions of the user's hands,said inputs for dynamically manipulating said selected user tool imageand/or the image of the tissues for dynamically interacting with therealistic image of the tissues during the simulated medical procedurefor display to the user on said display in real-time, wherein thedynamic interaction between the user tool image and the image of thetissues is displayed on said display using images with realistic visualfeatures exhibiting realistic mechanical interactions; and a userinterface providing a tool to adjust the dynamic image of the tissuesdisplayed on said display by adding or modifying features of saidtissues for display to compensate for anatomical structures that are inthe actual biological tissue of the particular patient but are missingfrom the dynamic image of tissues originally displayed such that thedynamic image of tissues displayed are subsequently displayed on thedisplay with the added or modified features, wherein the dynamicinteraction between the user tool image and the image of the tissues isdisplayed on said display providing realistic visual features exhibitingrealistic mechanical interactions based on the stored physicalcharacteristics.
 15. The modeling system of claim 14, wherein the toolto adjust the dynamic image of tissues includes a tool to provide theability to draw any geometric shape on the dynamic image of tissues. 16.The modeling system of claim 14, wherein the tool to adjust the dynamicimage of tissues includes a tool to provide the ability to complete anincomplete anatomical structure of the dynamic image of tissues.
 17. Themodeling system of claim 14, wherein the tool to adjust the dynamicimage of tissues includes a tool to provide the ability to modify thetexture, lighting, shadow and/or shading of a portion of the dynamicimage of tissues
 18. The modeling system of claim 14, wherein saidmedical images of the particular patient include an image of ananeurysm, and wherein said dynamic image includes an image of theaneurysm, and further wherein said user tool includes an aneurysm clipapplier for applying an aneurysm clip model for dynamically interactingwith the image of tissues.
 19. The modeling system of claim 14, whereinthe tool to adjust the dynamic image of tissues includes a tool toprovide the ability to command a tool to interact with one or moreportions of the dynamic image of tissues.
 20. The modeling system ofclaim 14, wherein the tool to adjust the dynamic image of tissuesincludes a tool to provide the ability to select elements the tool modeland/or the dynamic image of tissues for removal from the displayedimage.
 21. The modeling system of claim 14, wherein the tool to adjustthe dynamic image of tissues includes a tool to provide the ability toreposition objects in the displayed image by selecting the objects anddragging the objects to a desired position for display in the image. 22.The modeling system of claim 14, further comprising: a database forstoring a library of a plurality of models of different implants; and auser interface for selecting one implant model from said plurality ofmodels for use with said user tool model for dynamically interactingwith said image of tissues.
 23. The modeling system of claim 14, whereina tool to adjust the dynamic image of tissues includes tools to providethe abilities to: draw any geometric shape on the dynamic image oftissues; complete an incomplete anatomical structure of the dynamicimage of tissues; or modify the texture, and/or lighting of a portion ofthe dynamic image of tissues.
 24. The modeling system of claim 14,further comprising a user interface that can track the motions of anactual surgical instrument being used by the user with the particularpatient, such that said motions are used for dynamically manipulatingsaid selected user tool image and/or the image of the tissues fordynamically interacting with the realistic image of the tissues duringthe simulated medical procedure for display to the user on said displayin real-time.
 25. The modeling system of claim 24, wherein said userinterface that can track the motions of an actual surgical instrumentincludes a GPS receiver, an accelerometer, a magnetic detection device,or a camera.
 26. A modeling system for enabling a user to perform asimulated medical procedure, said system comprising: one or morecomputers; a display for displaying images to the user; an imagegenerator using one or more of said computers for executing software forgenerating a dynamic realistic image of the tissues for particularpatient for displaying on said display, wherein said realistic image ofthe tissues is provided showing an appearance including shadowing andtextures indicative of actual tissues; a database for storing a usertool library for providing a plurality of user tool models of actualuser tools used in medical procedures; a user interface for acceptinginputs from the user for selecting one of the user tool models; a usertool generator using one or more of said computers for executingsoftware for generating a realistic tool image of the selected user toolmodel for displaying on said display; and a user interface that cantrack the motions of an actual surgical instrument being used by theuser with the particular patient, such that said motions are used fordynamically manipulating said selected user tool image and/or the imageof the tissues for dynamically interacting with the realistic image ofthe tissues during the simulated medical procedure for display to theuser on said display in real-time. GPS, accelerometers, magneticdetection, or other location and motion detecting devices.
 27. Themodeling system of claim 26, wherein said user interface that can trackthe motions of an actual surgical instrument includes a GPS receiver, anaccelerometer, a magnetic detection device, or a camera.
 28. Themodeling system of claim 26, further comprising a user interfaceincluding a camera for accepting inputs from the user based on motionsof the user's hands, said inputs for dynamically manipulating saidselected user tool image and/or the image of the tissues for dynamicallyinteracting with the realistic image of the tissues during the simulatedmedical procedure for display to the user on said display in real-time,wherein the dynamic interaction between the user tool image and theimage of the tissues is displayed on said display using images withrealistic visual features exhibiting realistic mechanical interactions.